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DETAILED ACTION 
Response to Amendment 
1. The preliminary amendment filed on 03/05/2003 has been entered. Claims 1,6, 11, 14, 
18 and 22 have been amended. Claims 23-25 have been canceled. 



Claim Rejections - 35 USC § 102 

1. The following is a quotation of the appropriate paragraphs of 35 U.S. C 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an apphcation for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international apphcation filed under the treaty defmed in section 35 1(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

2. Claims 1-22 are rejected under 35 U.S.C. 102(e) as being anticipated by Xiong et al. U.S. 
Patent No. 6,434,265. 

3. Claim 1: 

U.S. Pat. No. 6,434,265 to Xiong teaches an image synthesis method comprising: 
A placement information obtaining step of obtaining placement information about a 
plurality of images in which adjacent images have a common subject region (e.g., in column 4, 
lines 5-40, Xiong teaches a method for constructing a panorama from rectilinear images in 3D 
through projective registration and calibration including: (1) the projective registrations of 
overlapping images, (2) calibration and global optimization of these images, a self calibration in 
which 2D image planes are positioned as 3D planes in space); 
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A setting step of setting one mapping mode out of a plurality of mapping modes each 
corresponding to a different mapping surface in accordance with said obtained placement 
information (e.g., in column 8, lines 18-58 of Xiong, it is stated that "overlapping photographs 
are analyzed to determine what orientation the photographs were taken in order to establish a 
common ground for subsequent operations and the panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane for viewing". The Xiong discloses some typical geometry on which panoramas are 
formed; In column 8, lines 18-58, Xiong further discloses that panorama is constructed on a 
particular geometry that will best facilitate the rendering of the projection onto a chosen viewing 
plane such as cubic, polyhedral, cylindrical and spherical geometries); and 

A synthesis step of combining said plurality of images by using the mapping mode set in 
said setting step (e.g., in column 4, lines 5-40 of Xiong, it is stated that the composing or 
blending in which images are ready to be re-projected to a 3D environment map with pixels in 
overlap regions being composed from multiple; In column 8, lines 18-58, Xiong further teaches 
that overlapping photographs are analyzed to determine what orientation the photographs were 
taken in order to estabHsh a common ground for subsequent operations and the panorama is 
constructed on a particular geometry that will best facilitate the rendering of the projection of the 
panorama onto a chosen viewing plane for viewing. The Xiong further discloses some typical 
geometry on which panoramas are formed). 

- The Examiner interprets "a placement information obtaining step of obtaining placement 
information about a plurality of images in which adjacent images have a common subject 
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region" as an automatic registration and calibration step of registering the overlapping 
images and capturing common overlapping areas between overlapping images and 
minimizing the average squared pixel intensity difference with respect to certain 
transformation parameters. 
- The Examiner interprets the mapping mode as mapping images onto a geometric surface 
such as a planar or a cylindrical surface (Applicant's specification, page 1, lines 20-25). 
Accordingly, the Examiner interprets the setting step of setting a mapping mode as the 
selecting step of selecting a geometric surface. In column 8, lines 18-58, Xiong discloses 
that panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection onto a chosen viewing plane (a geometric surface) such as 
cubic, polyhedral, cylindrical and spherical geometries. In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong fulfills the 
claimed limitation of a setting step of setting one mapping mode out of a plurality of 
mapping modes each corresponding to a different mapping surface in accordance with 
said obtained placement information. 



Claim 2: 
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Claim 2 recites all the limitations of claim 1 and adds the limitation of "a focal length 
obtaining step." The Xiong reference teaches finding projective parameters such as 3D rotation 
parameters (pan, tih roll), center of projection of images, ratio of focal lengths, and the like 
(column 10, hnes 9-28). The Xiong reference implicitly teaches finding the camera internal 
parameters vector of which the focal length is a component (column 1 1, lines 15-42). 

Claim 3: 

The claim 3 recites all the limitations of claim 1 and adds the hmitation of "a changing 
step of changing the mapping mode." The Xiong reference teaches that panorama is constructed 
on a particular geometry that will best facilitate the rendering of the projection onto a chosen 
viewing plane such as cubic, polyhedral, cylindrical and spherical geometries (column 8, lines 
18-58). 

Claim 4: 

The claim 4 recites all the Umitations of claim 1 and adds the hmitation of "issuing a 
warning." The Xiong reference teaches in figures 2 and 3 a user interface and a global 
optimization that provides feedback to the computer system such as issuing warning messages on 
the computer monitor 218 when the pair- wise objective function is not desirable for a poor 
selection of the projection viewing plane (and the resulting panoramas will have imperfectly 
aligned images that give shadow or ghosting effects) due to a poor selection of the projection 
viewing plane. The resulting panoramas will have imperfectly aligned images that give shadow 
or ghosting effects. Nevertheless, the Xiong reference points to a seamless multi-resolution 
average blending method that would result in an absent of shadow effects (column 14, lines 1- 
45). 
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Claim 5: 

The claim 5 recites all the hmitations of claim 1 and adds the Umitation of "a displaying 
step of displaying a cuttable rectangular region." The Xiong reference teaches how to align 
images more precisely by changing the coordinates for positioning an image. The Xiong 
reference further teaches placing the images 1210 at selected tangents on the viewing sphere 
1220 (figure 12, and column 17, lines 12-65). 

4. Claim 6: 

The claim 6 encompasses the same scope of invention as that of claim 1 except additional 
claimed limitation of "an image synthesis apparatus. However, Xiong et al. further discloses in 
figure 3 an apparatus for image synthesis. 

Claim 7: 

The claim 7 recites all the hmitations of claim 6 and adds the Umitation of "a focal 
length obtaining step." The Xiong reference teaches finding projective parameters such as 3D 
rotation parameters (pan, tilt roll), center of projection of images, ratio of focal lengths, and the 
like (column 10, lines 9-28). The Xiong reference implicitly teaches finding the camera internal 
parameters vector in his image synthesis apparatus of figure 3 because the focal length is a 
component of that vector (column 11, lines 15-42). 

Claim 8: 

The claim 8 recites all the limitations of claim 6 and adds the limitation of "a changing 
step of changing the mapping mode." The Xiong reference teaches that panorama is constructed 
on a particular geometry that will best facilitate the rendering of the projection of the panorama 
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onto a chosen viewing plane such as cubic, polyhedral, cylindrical and spherical geometries 
(column 8, lines 18-58). 
Claim 9: 

The claim 9 recites all the limitations of claim 8 and adds the limitation of "issuing a 
warning." The Xiong reference teaches in figures 2 and 3 a user interface and a global 
optimization that provides feedback to the computer system such as issuing warning messages on 
the computer monitor 218, e.g., when the pair-wise objective function is not desirable (and the 
resulting panoramas will have imperfectly aligned images that give shadow or ghosting effects) 
due to a poor selection of the projection viewing plane. The resulting panoramas will have 
imperfectly aligned images that give shadow or ghosting effects. Nevertheless, the Xiong 
reference points to a seamless multi -resolution average blending that is absent of shadow effects 
(column 14, lines 1-45). 

Claim 10: 

The claim 10 recites all the limitations of claim 8 and adds the limitation of "a displaying 
step of displaying a cuttable rectangular region." The Xiong reference teaches how to align 
images more precisely by changing the coordinates for positioning an image. The Xiong 
reference further teaches placing the images 1210 at selected tangents on the viewing sphere 
1220 (figure 12, and column 17, lines 12-65). 

5. Claim 11: 

The claim 1 1 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program for 
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implementing image synthesis method. However, Xiong et al. further discloses the claimed 
limitation of a computer-readable storage medium having a program for implementing image 
synthesis method (e.g., in column 3, lines 54-57, it is stated "a program residing in system 
memory 220 which stores output data and other data"). 
Claim 12: 

The claim 12 recites all the Umitations of claim 1 1 and adds "a focal length obtaining 
step." The Xiong reference teaches finding projective parameters such as 3D rotation parameters 
(pan, tilt roll), center of projection of images, ratio of focal lengths, and the like (column 10, 
lines 9-28). The Xiong reference implicitly teaches finding the camera internal parameters vector 
of which the focal length is a component (column 11, lines 15-42). 

Claim 13: 

The claim 13 recites all the Umitations of claim 1 1 and adds the limitation of "a changing 
step of changing the mapping mode." The Xiong reference teaches that panorama is constructed 
on a particular geometry that will best facilitate the rendering of the projection of the panorama 
onto a chosen viewing plane such as one of the cubic, polyhedral, cylindrical and spherical 
geometries (column 8, lines 18-58). 

6. Claim 14: 

The claim 14 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of an image synthesis method comprising a generating step of 
generating coordinate-space transformation parameters. However, Xiong et al. further discloses 
the claimed limitation of a generating step of generating coordinate-space transformation 
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parameters (for example, in a local pair-wise registration, the estimation of parameters in a 
projective matrix which represents a coordinate transformation matrix; column 1 1 , lines 1-46). 
Claim 15: 

The claim 15 recites all the Hmitations of claim 14 and adds the limitation of "a changing 
instruction step of issuing a mapping mode changing instructions." The Xiong reference teaches 
that panorama is constructed on a particular geometry that will best facilitate the rendering of the 
projection of the panorama onto a chosen viewing plane such as cubic, polyhedral, cylindrical 
and spherical geometries (column 8, lines 18-58). The Xiong reference further teaches 
calibration as the second major step in authoring panoramas to extract camera internal and 
external parameters from those projective matrices (column 12, lines 33-44). The Xiong 
reference discloses that the improved projective matrix parameters will in turn be used to 
generate improved estimation of camera parameters and blending may be further iterated after an 
initial blending (column 13, lines 53-67). 

Claim 16: 

The claim 16 recites an image synthesis method comprising a reference position-setting 
step. The Xiong reference teaches a multi-resolution weighted average blending for perspective 
alignment (column 14, lines 2-45). The Xiong reference also teaches for example in column 1 1 
setting or estimating camera orientation and the rotation matrix that is based on camera 
orientation. 

Claim 17: 

The claim 17 recites all the hmitations of claim 14 and adds the limitation of "a storage 
step of storing a generated panoramic synthesized image." The Xiong reference teaches 
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panoramic canvas which can be simply be a buffer or memory in a computer system such as 
memory 220 of figure 2 and onto which the blended images is copied (column 15, hnes 15-28). 
7. Claim 18: 

The claim 18 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of an image synthesis apparatus comprising a generating means for 
generating coordinate- space transformation parameters and image synthesis means for 
combining the images. However, Xiong et al. further discloses the claimed limitation of an 
image synthesis apparatus comprising a generating means for generating coordinate-space 
transformation parameters and image synthesis means for combining the images (e.g., a pair- 
wise registration module 222 and the projection function module 228. The parameters in a 
projective matrix are estimated which represents an image coordinate transformation matrix. The 
projection function module 228 constructs a panoramic scene by projecting the blended image 
onto any designated geometry view surface; column 4, lines 21-52, and column 1 1, lines 1-46). 

Claim 19: 

The claim 19 recites all the limitations of claim 18 and adds the limitation of "a changing 
means for changing the mapping mode." The Xiong reference teaches a projection module 228 
residing in memory 220 and operating the processor 212 may be controlled through the user 
interface 230 to allow a user to select what geometry will be projected onto (i.e., changing the 
mapping mode) and to control and modify other factors, including the use of photo re-touching 
software such as PhotoShop for modifying the final panorama (column 4, Unes 41-49). 

Claim 20: 
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The claim 20 recites all the hmitations of claim 18 and adds the limitation of "a reference 
position setting means." The Xiong reference teaches for example in column 1 1 setting or 
estimating camera orientation ,on which the rotation matrix is based. 

Claim 21: 

The claim 21 recites all the hmitations of claim 18 and adds the limitation of "storage 
means for storing a generated panoramic synthesized image, coordinate transformation 
parameters and coordinate-space transformation parameters." The Xiong reference teaches 
storage means such as panoramic canvas which can be simply a buffer or memory in a computer 
system such as memory 220 in figure 2 and onto which the blended images is copied (column 15, 
lines 15-28). The Xiong reference teaches that the improved projective matrix parameters (i.e., 
the coordinate and coordinate-space transformation parameters) can be used to generate 
improved estimation of camera parameters and blending may be further iterated after an initial 
blending (column 13, lines 53-67). Finally, the Xiong reference teaches projective parameters, 
e.g., 3D rotation parameters, center of projection of images, ratio of focal lengths, and an afFme 
transformation in constructing a two-dimensional view of the environment (column 10, lines 16- 
28). 

8. Claim 22: 

The claim 22 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program for 
implementing image synthesis method and a generating step of generating coordinate-space 
transformation parameters. However, Xiong et al. further discloses the claimed limitation of a 
computer-readable storage medium having a program for implementing image synthesis method 
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and a generating step of generating coordinate- space transformation parameters (e.g., a program 
residing in system memory 220 which stores output data and other data; column 3, lines 54-67. 
The Xiong reference also teaches in a local pair-wise registration the estimation of parameters in 
a projective matrix which represents a coordinate transformation matrix; column 11, lines 1-46. 
Finally, the Xiong reference teaches that the improved projective matrix parameters will in turn 
be used to generate improved estimation of camera parameters and blending may be further 
iterated after an initial blending; column 13, lines 53-67). 

Remarks 

9. Applicant's arguments, filed 03/05/2003, paper number 6, have been fully considered but 
they are not deemed to be persuasive. 

10. Applicant argues in essence with respect to claim 1 and similar claims that: 

"In Xiong et al., an image is projected onto a predetermined surface for image synthesis, 
and the combined image is projected onto an arbitrary surface for image display. 
Therefore, in Xiong et al, the synthesized image is deteriorated by multiple projections. 
On the contrary, the method of Claim 1 makes it possible to decrease the number of 
unnecessary projections by providing flexibility in the projection at the time of image 
synthesis." 

This is not found persuasive for the following reasons: 

(a) Applicant's states that "In Xiong et al., an image is projected onto a predetermined 
surface for image synthesis". On contrary, Xiong teaches the projection of a panorama 
onto a particular 3D geometry (not a predetermined surface). In column 8, Unes 19-67, it 
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is stated that "Authoring a panorama from 2D images can be thought of as divided into 
two different phases: (1) orientation of originally 2D images into 3D space, and (2) the 
projection of a panorama onto a particular 3D geometry, that can later be used to project 
views of the panorama onto a 2D viewing plane". Applicant further came to conclusion 
that "therefore, in Xiong et al., the synthesized image is deteriorated by multiple 
projections". This is not found persuasive since the projection onto a 3D geometry would 
not deteriorate a synthesized image because the panorama is constructed on a particular 
geometry that will best facilitate the subsequent step of the projection of the panorama 
from the particular geometry onto a chosen viewing plane for viewing by a user. 
Therefore, Xiong' s teaching only involves one projection for mapping of the panorama 
from a 3D geometric geometry onto a 2D geometric surface. 



1 1 . Applicant argues in essence with respect to claim 1 and similar claims that: 

"Nothing has been found in Xiong et al. that would teach or suggest setting one mapping 
mode out of a plurality of mapping modes, each corresponding to a different mapping 
surface, in accordance with obtained placement information about a plurality of images." 

This is not found persuasive because of the following reasons: 

- The Examiner interprets the mapping mode as mapping images onto a geometric surface 
such as a planar or a cylindrical surface (Applicant's specification, page 1, lines 20-25). 
Accordingly, the Examiner interprets the setting step of setting a mapping mode as the 
selecting step of selecting a geometric surface. In column 8, lines 18-58, Xiong discloses 
that panorama is constructed on a particular geometry that will best facilitate the 
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rendering of the projection onto a chosen viewing plane (a geometric surface) such as 
cubic, polyhedral, cylindrical and spherical geometries. In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong fulfills the 
claimed limitation of a setting step of setting one mapping mode out of a plurality of 
mapping modes each corresponding to a different mapping surface in accordance with 
said obtained placement information. 



Conclusion 

12. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a), 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 . 136(a) will be calculated ft'om the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS fi-om the mailing 
date of this final action. 
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13. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (703) 605-1213. 
The examiner can normally be reached on 8:00 AM - 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers for the 
organization where this application or proceeding is assigned are (703) 308-6606 for regular 
communications and (703) 308-6606 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703) 395-3900. 



jcw 



March 13, 2003 
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